Determining user interface elements required for task completion using machine learning

ABSTRACT

Disclosed herein are system, method, and computer program product embodiments for determining the minimum set of user interface (UI) elements for a task. One embodiment operates by transmitting UI elements associated with an application to a first computing device operated by a first user and a second computing device operated by a second user. Further, the embodiment includes obtaining data structures corresponding to a first task and a second task and then grouping, using a machine learning model, the data structures into a first cluster corresponding to the first task and a second cluster corresponding to the second task. Further still, the embodiment also includes determining a minimum set of UI elements required for the first task based on the first cluster.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of U.S. Provisional PatentApplication Ser. No. 62/968,780, entitled: “Determining User InterfaceElements Required for Task Completion using Machine Learning,” whichfiled on Jan. 31, 2020. Accordingly, this application claims priorityunder 35 U.S.C. § 119(e) to U.S. Provisional Patent Application Ser. No.62/968,780. U.S. Provisional Patent Application Ser. No. 62/968,780 ishereby incorporated by reference in its entirety.

BACKGROUND

It is not uncommon for a software application to have one or moreworkflows to perform one or more tasks. During each workflow, the usermay be presented with a large number of user interface (UI) elements(e.g., GUI widgets) distributed across multiple screens. In order tocomplete a task, the user might only manipulate some of the UI elements,leaving other UI elements untouched. Moreover, different tasks mayrequire the user to manipulate different UI elements.

The large number of UI elements may increase the time needed by the userto complete the task because the user must determine which UI elementsneed to be manipulated. Further, the large number of UI elements mayresult in errors due to the user manipulating UI elements that do notneed to be manipulated to complete the task. Further still, the largenumber of UI elements may result in excess network bandwidth consumptionbecause UI elements that do not need to be manipulated by the user tocomplete the task are still transmitted to a user's computing device.Regardless, workflows and UI elements remain a popular way for user's tocomplete tasks within a software application.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings are incorporated herein and form a part of thespecification.

FIG. 1 shows a block diagram of a system in accordance with one or moreembodiments.

FIG. 2 shows a flowchart for determining a minimum set of UI elementsrequired to complete a task in accordance with one or more embodiments.

FIG. 3 shows an example of determining a minimum set of UI elementsrequired to complete a task in accordance with one or more embodiments.

FIG. 4 shows a flowchart for generating a custom UI for completing apredicted task in accordance with one or more embodiments.

FIG. 5 shows an example of a custom UI for completing a task along witha preview image of another custom UI for completing a different task inaccordance with one or more embodiments.

FIG. 6 is an example computer system useful for implementing variousembodiments.

In the drawings, like reference numbers generally indicate identical orsimilar elements. Additionally, generally, the left-most digit(s) of areference number identifies the drawing in which the reference numberfirst appears.

DETAILED DESCRIPTION

Provided herein are system, method, and computer program productembodiments that utilize machine learning to determine the minimum setof UI elements required to complete a task of an application.Specifically, data structures are used to record which UI elements arebeing manipulated by users, and then the data structures are groupedinto clusters by a machine learning model. Each cluster may correspondto a task. The minimum set of UI elements required to complete the taskis determined by performing a union operation on the data structures inthe cluster corresponding to the task. The clusters may be ranked andthe distances between pairs of clusters, representing the similaritybetween two tasks, may also be determined. A custom UI for the task maybe generated with the minimum set of UI elements, but not the other UIelements, and transmitted to user's computing device to complete thetask. As discussed throughout this disclosure, the disclosed embodimentsreduce task completion time, reduce task errors, and reduce bandwidthconsumption over existing systems and processes.

Also provided herein are system, method, and computer program productembodiments that determine and rank tasks that the user is trying tocomplete based on contextual information. A custom UI for eachdetermined task may be generated. Each custom UI has the minimum numberof UI elements required to complete the corresponding task. These customUIs may be transmitted to a user device operated by the user. Images ofone or more of the custom UIs, instead of the custom UIs themselves, mayalso be generated and transmitted to the user device as previews for thecustom UIs.

FIG. 1 shows a system 100 in accordance with one or more embodiments. Asshown in FIG. 1, the system 100 has multiple components including, forexample, an event listener 102, a machine learning model 104, a UIgenerator 106, a cluster management engine 108, a prediction model 180,and a repository 160. Each of these components (102, 104, 106, 108, 160,180) may execute on the same computing device (e.g., server, mainframe,desktop personal computer (PC), laptop, tablet PC, smartphone, etc.) oron different computing devices connected by a network having wiredand/or wireless segments. These components (102, 104, 106, 108, 120,180) may form at least part of an application (e.g., email application,customer relationship management application, social media application,image processing application, word processing application, encryptionapplication, web browser, etc.). Moreover, a portion of the applicationmay reside and/or execute on the computing device of a user (e.g., userdevice 1 122A, user device X 122X). The user devices 122 may correspondto desktop PCs, laptops, tablet PCs, smartphones, etc.

The application is associated with multiple UI elements 168 (e.g., UIelement 1 168A, UI element Z 168Z). Each UI element 168 may correspondto a textbox, a set of radio buttons, a dropdown list, a slider, or anyGUI widget that can be manipulated by a user. The UI elements 168 may betransmitted to user devices 122 and then displayed across multiplescreens. In order to complete different tasks, users may be required tomanipulate different subsets of the UI elements 168. Example tasksinclude updating records, configuring the options on an outgoing email,configuring user permissions, creating or editing steps in a workflow,updating a product catalog, creating new records, configuring databasefields, creating or editing components inside of an applicationinterface builder etc. Accordingly, manipulating the UI elements 168 mayupdate fields in tables, configure the settings on an outgoing email,update a database field or schema, execute a process on another machine(aka a server), schedule a process to be executed later etc.

In one or more embodiments, the event listener 102 is configured totrack which UI elements 168 are being manipulated by a user while theuser is completing a task within the application. The event listener 102may record the identities of the manipulated UI elements in a datastructure 162 (e.g., vector, linked list, array, etc.). Additionally oralternatively, the manipulated UI elements themselves could beinstrumented to update the data structure 162. For example, datastructure 1 162A may include the identities of the UI elementsmanipulated while a user was completing a task. Another data structure(not shown) may include the identities of the UI elements manipulatedwhile a different user was completing the same task. Data structure N162N may include the identities of the UI elements manipulated while thesame user or a different user was completing a different task. Theidentities of the UI elements in a data structure may be sorted based onthe order in which the UI elements appears in the workflow. For example,in the case of a multi-screen workflow, the identity of a UI elementthat appeared on the first screen would come before the identity of a UIelement that appeared on the last screen. Similarly, the identity of aUI element that appeared on a top of a screen would come before theidentity of a UI element that appeared on the bottom of the same screen.Additionally or alternatively, the order in which the manipulated UIelements appeared in the workflow may be stored in the data structure162 separately from the identities of the manipulated UI elements.Additionally or alternatively, the order in which the user manipulatedthe UI elements may be stored in the data structure. The identity of aUI element may appear more than once in a data structure 162 if the usermanipulated the UI element multiple times to complete the task. In oneor more embodiments, the identity of the screen on which the manipulatedUI element appeared (e.g., screen 3 of the 5 screen workflow) is alsostored in the data structure.

In one or more embodiments, the event listener 102 stores a timestamp164 associated with each data structure 162. The timestamp 164 reflectsthe time (e.g., year, month, day, hour, minute, second, etc.) when thetask associated with the data structure 162 was completed. For example,timestamp 1 164A reflects the time when the task associated with datastructure 1 162A was completed. Similarly, timestamp N 164N reflects thetime when the task associated with the data structure N 162N wascompleted.

In one or more embodiments, the machine learning model 104 is configuredto group the data structures 162 into clusters based on the UI elementsidentified in each of the data structures 162. The machine learningmodel 104 may utilize any type of clustering algorithm (e.g.,centroid-based clustering, density-based clustering, hierarchicalclustering, distribution-based clustering, etc.) to group the datastructures 162. Each of the clusters may correspond to one of the tasks.

In one or more embodiments, the cluster management engine 108 isconfigured to calculate age statistics for the clusters. For example,the age statistic of a cluster may correspond to the average age of thedata structures in the cluster, as determined from the timestamps 164.As another example, the age statistic of a cluster may correspond to theage of the youngest data structure in the cluster or the standarddeviation of the ages of the data structures in the cluster, asdetermined from the timestamps 164.

In one or more embodiments, the cluster management engine 108 isconfigured to calculate distances between pairs of clusters. Forexample, the distance between two clusters may be the distance from thecenter of one cluster to the center of the other cluster. A smalldistance is indicative of the two tasks corresponding to the twoclusters being similar. Similarly, a large distance is indicative of thetwo tasks corresponding to the two clusters being unrelated. Thesedistances (not shown) may also be stored in the repository 160. Theclosest cluster (i.e., the cluster that is the shortest distance away)may be referred to as the nearest neighbor.

In one or more embodiments, the prediction model 180 is configured todetermine one or more tasks in the application that the user isattempting to complete. The prediction model 180 may implement machinelearning to determine one or more tasks in the application that the useris attempting to complete. The prediction model 180 may determine theone or more tasks based on contextual information associated with theapplication and the user. For example, contextual information mayinclude the time, day, week, quarter, beginning/end of quarter when theuser is accessing the application. As another example, contextualinformation may include the objects (e.g., tables, variables, files,etc.) the user most recently interacted with, objects created by theuser that have changed since the user's last use of the application,objects with timestamps that are close to the current time/date. Asanother example, contextual information may include recent searchqueries requested by the user. As another example, contextualinformation may include the device the user is using (a Desktop PC,smartphone, tablet, etc.), the available bandwidth of their networkconnection (e.g. a 3G cellular connection, or a Gigabit fiberconnection), or their geography either absolute, or relative to thecomputing device sending the UI. As yet another example, contextualinformation may include other users who have employment positions ortitles similar to the position/title of the user, other users who theuser follows (e.g., on social media), etc.

In one or more embodiments, the prediction model 180 is configured torank the one or more determined tasks. The task with the highest rankcorresponds to the task the user is most likely attempting to completeaccording to the prediction model 180. The remaining determined taskshave lower rankings (i.e., the user is less likely to be attempting tocomplete the remaining tasks, according to the prediction model 180). Asdiscussed above, each task may correspond to a cluster. The rank of atask may be based on the cardinality of the corresponding cluster (i.e.,number of data structures 162 in the cluster) relative to thecardinality of the other clusters (corresponding to other tasks). Forexample, the more data structures in a cluster, the higher the rankingof the corresponding task. Additionally or alternatively, the rank of atask may depend on an age statistic of the task's cluster relative tothe age statistics of the other clusters (corresponding to other tasks).A task corresponding to a cluster with young data structures may beranked higher than a task corresponding to a cluster with older datastructures.

As discussed above, each task may correspond to a cluster and distancesbetween clusters may be calculated. In one or more embodiments, at leastsome of the rankings may also be based on these distances. For example,assume cluster A, cluster B, and cluster C correspond to task A, task B,and task C, respectively. Moreover, assume the distance between clusterA and cluster C is shorter than the distance between cluster A andcluster B. If it is determined, based on contextual information, thatthe user is most likely attempting to complete task A, task A will havethe highest rank. Task C will be assigned the next highest rank becausecluster C is the nearest neighbor of cluster A corresponding to task A.Task B will be assigned a rank that is lower than the rank of task Cbecause cluster B is further away from cluster A than cluster C.Although only three tasks/clusters are mentioned, this ranking schemebased on distance may be used for any number of tasks/clusters.

The rankings 166 associated with the tasks may also be stored in therepository 160. For example, ranking 1 166A and ranking Q 166Qcorrespond to the rankings of task 1 and task Q (not shown),respectively.

In one or more embodiments, the UI generator 106 is configured todetermine the minimum set of UI elements required for a task based onthe cluster corresponding to the task. Specifically, the UI generatormay perform a union operation on all the UI elements identified in alldata structures of the cluster. The result of this union operation isthe minimum set of UI elements required for the task corresponding tothe cluster. For example, minimum set of UI elements 1 169A and minimumset of UI elements Q 169Q correspond to the minimum set of UI elementsfor task 1 and the minimum set of UI elements for task Q, respectively.The union operation may preserve the sorting of the identities of the UIelements (discussed below). Accordingly, UI elements in a minimum setare ordered as the UI elements would have been in the originalmulti-screen workflow (i.e., a UI element on the first screen comesbefore a UI element on the last screen, a UI element on the top of thescreen comes before a UI element on the bottom of the screen, etc.).

Those skilled in the art having the benefit of this disclosure willappreciate that during the union operation two data structures mighthave conflicting ordering information for the UI elements. In suchembodiments, the newer data structure, as determined from thetimestamps, controls the ordering. For example, if there are two UIelements in the union from two different data structures that have thesame visual/interactive order, then the UI element from the newer datastructure would be “above” or “before” the UI element from the olderdata structure. All subsequent UI element orderings may be adjustedaccordingly.

In one or more embodiments, the UI generator 106 is configured togenerate a custom UI for a task (e.g., the task with the highest rankingas determined by the prediction model 180) based on the minimum set ofUI elements 169 required to complete the task. The custom UI includesthe minimum set of UI elements for the task, but may exclude all theother UI elements 168 associated with the application. Unlike themulti-screen workflow, the custom UI may be a single screen. The minimumset of UI elements may be positioned in the custom UI based on theordering of the UI elements in the minimum set. As discussed above, thisis consistent with the order in which the UI elements appeared in theoriginal multi-screen workflow. In one or more embodiments, since thecustom UI only includes the minimum set of UI elements (and not all theUI elements 168 associated with the application), the custom UI can fiton a single screen. The UI generator 106 may transmit the custom UI to auser device operated by a user so that the user may complete the task.

As discussed above, the identity of the screen on which the manipulatedUI element appeared (e.g., screen 3 of the 5 screen workflow) may alsobe stored in the data structure. In one or more embodiments, in theevent the custom UI does not fit on a single screen, the identity of thescreens, as recorded in the data structures, may be used as anadditional heuristic to consolidate the minimal set of elements onto aminimal set of screens. The minimal set of screens would still be fewerscreens than the original workflow.

In one or more embodiments, the UI generator 106 is configured togenerate an image of a custom UI for a task. The image may have a lowresolution in order to reduce the storage space needed to store theimage and/or reduce the network bandwidth needed to transmit the image(e.g., to user devices 122). However, the resolution must be high enoughto allow for the perception of the gestalt of the task (e.g., theresolution might need to be sufficiently high for the viewer (e.g.,user) to be able to identify one or more of the UI elements present inthe custom UI).

In one or more embodiments, the UI generator 106 is configured togenerate options for display with the custom UI. These options maycorrespond to images of other custom UIs. These options are selectableby the user. For example, the user selection of an option triggers thecorresponding image of another custom UI to be displayed (discussedbelow).

FIG. 2 shows a flowchart in accordance with one or more embodiments. Theflowchart of FIG. 2 depicts a process for determining the minimum set ofUI elements required to complete a task. One or more of the steps inFIG. 2 may be performed by the components (102, 104, 106, 108) discussedabove in reference to FIG. 1. In one or more embodiments, one or more ofthe steps shown in FIG. 2 may be omitted, repeated, and/or performed ina different order than the order shown in FIG. 2. Accordingly, the scopeof the invention should not be considered limited to the specificarrangement of steps shown in FIG. 2.

In Step 205, the UI elements associated with an application aretransmitted to user devices and displayed across multiple screens aspart of a multi-screen workflow. Each UI element may correspond to atextbox, a set of radio buttons, a dropdown list, a slider, or any GUIwidget that can be manipulated by a user. In order to complete differenttasks in the application, users operating the user devices may berequired to manipulate different subsets of the UI elements. However,depending on the task in the application a user is completing, some ofthe UI elements will not be touched by the user. Accordingly, differentsubsets of the UI elements are manipulated by users to completedifferent tasks.

In Step 210, data structures are obtained. Each data structure may be avector, an array, a linked-list, etc. Each data structure identifies thesubset of the UI elements that were manipulated by a user to complete atask of the application. While the user is completing the task, theuser's actions may be monitored (e.g., tracked) to determine which UIelements are being manipulated and which UI elements are not beingtouched by the user. In one or more embodiments, for privacy reasons,only the user's manipulations of the UI elements (e.g., not keystrokes)would be monitored. The identities of the modified UI elements may bestored in the data structures. Different data structures may correspondto different completed tasks and/or to different users. The identitiesof the UI elements in a data structure may be sorted based on the orderin which the UI elements appear in the multi-screen workflow (i.e., theidentity of a UI element on the first screen comes before the identityof a UI element on the last screen, the identity of a UI element on thetop of the screen comes before the identity of a UI element on thebottom of the screen, etc.). A timestamp may be associated with eachdata structure. The timestamp may reflect the time when the taskcorresponding to the data structure was completed. A timestamp may begenerated and stored for each interaction with a UI element as well.

Those skilled in the art, having the benefit of this disclosure, willappreciate that it is possible for different users completing the sametask to manipulate slightly different subsets of UI elements.Accordingly, multiple data structures corresponding to the same task mayidentify slightly different subsets of the UI elements. These variationsin the manipulated UI elements result from some UI elements beingoptional to complete a task.

In Step 215, the data structures are grouped into clusters using amachine learning model. For example, the data structures may be groupedusing centroid-based clustering, density-based clustering, hierarchicalclustering, distribution-based clustering, etc. Clustering effectivelyidentifies, based on the manipulated UI elements identified in the datastructures, which data structures correspond to the same task. Asdiscussed above, the data structures corresponding to the same task mayidentify slightly different subsets of UI elements due to themanipulation of some UI elements being optional.

In Step 220, a cluster of data structures is selected. The cluster isselected from clusters that have not yet been selected. For example, thecluster may be selected at random. As another example, the cluster withthe greatest cardinality (i.e., the greatest number of data structures)may be selected. As yet another example, the cluster with the smallestcardinality may be selected. As discussed above, each clustercorresponds to a task.

In Step 225, the minimum set of UI elements required to complete thetask corresponding to the selected cluster is determined. In one or moreembodiments, determining the minimum set of UI elements includesexecuting a union operation on the subsets of UI elements identified inall the data structures in the selected cluster. The result of thisunion operation is the minimum set of UI elements required to completethe task corresponding to the selected cluster. The union operation maypreserve the sorting of the identities of the UI elements (discussedbelow). Accordingly, UI elements in a minimum set are ordered as the UIelements would have been in the original multi-screen workflow (i.e., aUI element on the first screen comes before a UI element on the lastscreen, a UI element on the top of the screen comes before a UI elementon the bottom of the screen, etc.).

In Step 230, it is determined if there remains one or more clusters thathave not yet been selected. When it is determined that at least oneunselected cluster remains, the process returns to Step 220. When it isdetermined that all of the clusters have been selected, and thus theminimum sets of UI elements required to complete all the taskscorresponding to the clusters have been determined, the process proceedsto Step 235.

In Step 235, distances between pairs of clusters are determined. Forexample, the distance between two clusters may be the distance from thecenter of one cluster to the center of the other cluster. A smalldistance is indicative of the two tasks corresponding to the twoclusters being similar. Similarly, a large distance is indicative of thetwo tasks corresponding to the two cluster being unrelated. Step 235 maybe optional.

Still referring to Step 235, the age statistics of the clusters aredetermined. For example, the age statistic of a cluster may correspondto the average age of the data structures in the cluster, as determinedfrom the timestamps. As another example, the age statistic of a clustermay correspond to the age of the youngest data structure in the clusteror the standard deviation of the ages of the data structure in thecluster, as determined from the timestamps.

Following Step 235, a custom UI may be generated to complete a task inthe application. The custom UI includes the minimum UI elements requiredto complete the task, as determined in Step 225, but excludes the otherUI elements associated with the application. The minimum set of UIelements may be positioned in the custom UI based on the ordering of theUI elements in the minimum set. As discussed above, this ordering isconsistent with the order in which the UI elements appeared in theoriginal multi-screen workflow. The custom UI may be transmitted to auser device for display so that the user operating the user device maymanipulated the UI elements and complete the task.

Those skilled in the art, having the benefit of this detaileddescription, will appreciate that because the custom UI only has theminimum set of UI elements required to complete the task, and not allthe UI elements, less network bandwidth is required to transmit thecustom UI to the user devices. Further, because custom UI only has theminimum set of UI elements, the user wastes less time determining whichUI elements need to be manipulated and thus the task can be completedmore quickly. Further still, because the custom UI only has the minimumset of UI elements, there is likely to be fewer errors in completing thetask because the user is not manipulating UI elements that should not betouched to complete the task.

FIG. 3 shows an example in accordance with one or more embodiments.Specifically, FIG. 3 shows an example of the process depicted in FIG. 2.As shown in FIG. 3, there are multiple data structures: data structure 1302A, data structure 2 302B, data structure 3 302C, data structure 4304A, and data structure 5 304B. Each of these data structurescorresponds to a task. Moreover, each of these data structures includesthe identities of UI elements that were manipulated to complete a task.For example, data structure 2 302B includes the identities of UIelements A, B, and C. UI elements A, B, and C were manipulated by a userto complete a task in the application. As another example, datastructure 5 304B includes the identities of UI elements B, C, and X304B. UI elements B, C, and X were manipulated by the same or adifferent user to complete a task in the application. If data structure2 302B and data structure 5 304B correspond to the same user, these twodata structures might be from two different sessions and thus havedifferent timestamps. If these two data structures correspond to twodifferent users, these two data structures might have the sametimestamps (i.e., the two users completed their respective tasks at thesame time) or different timestamps.

The data structures have been grouped into either cluster A 300A orcluster B 300B using a machine learning model. As shown, data structure1 302A, data structure 2 302B, and data structure 3 302C belong tocluster A 300A, while data structure 4 304A and data structure 5 304Bbelong to cluster B 300B. Data structures belonging to the same clusterare deemed to correspond to the same task.

Assume cluster A 300A corresponds to task A 398 in the application. Inorder to determine the minimum set of UI elements required for task A398, a union operation is performed on all the data structures 302 incluster A 300A. The result of the union operation, and thus the minimumset of UI elements required to complete task A 398, is UI elements A, B,C, D, and E (as shown in FIG. 3). In order to complete task A 398, acustom UI with UI elements A, B, C, D, and E may be generated andtransmitted to a user device. The user operating the user device maymanipulate UI elements A, B, C, D, and E to complete task A 398.

Assume cluster B 300B corresponds to task B 399 in the application. Inorder to determine the minimum set of UI elements required for task B399, a union operation is performed on all the data structures 304 incluster B 300B. The result of the union operation, and thus the minimumset of UI elements required to complete task B 399, is UI elements B, C,X, and Y (as shown in FIG. 3). In order to complete task B 399, a customUI with UI elements B, C, X, and Y may be generated and transmitted to auser device. The user operating the user device may manipulate UIelements B, C, X, and Y to complete task B 399.

FIG. 4 shows a flowchart in accordance with one or more embodiments. Theflowchart of FIG. 4 depicts a process for generating a custom UI forcompleting a predicted task. One or more of the steps in FIG. 4 may beperformed by the components (102, 104, 106, 108, 180) discussed above inreference to FIG. 1. One or more of the steps in FIG. 4 may be performedafter the steps in FIG. 2. In one or more embodiments, one or more ofthe steps shown in FIG. 4 may be omitted, repeated, and/or performed ina different order than the order shown in FIG. 4. Accordingly, the scopeof the invention should not be considered limited to the specificarrangement of steps shown in FIG. 4.

In Step 405, contextual information associated with a user operating theapplication on a user device is obtained. For example, contextualinformation may include the time, day, week, quarter, beginning/end ofquarter when the user is accessing the application. As another example,contextual information may include the objects (e.g., tables, variables,files, etc.) the user most recently interacted with, objects created bythe user that have changed since the user's last use of the application,objects with timestamps that are close to the current time/date. Asanother example, contextual information may include recent searchqueries requested by the user. As yet another example, contextualinformation may include other users who have employment positions ortitles similar to the position/title of the user, other users who theuser follows (e.g., on social media), etc.

In Step 410, one or more tasks the user is attempting to complete aredetermined based on the contextual information. These tasks areessentially predictions executed by a prediction model based on thecontextual information. The tasks may also be ranked. The task with thehighest rank corresponds to the task the user is most likely attemptingto complete according to the prediction model. The remaining determinedtasks have lower rankings (i.e., the user is less likely to beattempting to complete the remaining tasks, according to the predictionmodel).

As discussed above, each task may correspond to a cluster. As alsodiscussed above, the cardinality of each cluster, an age statistic foreach cluster, and distances between clusters may be calculated. Theranking of a task may depend on the cardinality of the correspondingcluster, the age statistic for the corresponding cluster, and/or thedistances between the corresponding cluster and other clusters(corresponding to other tasks).

In Step 415, the minimum set of UI elements required to complete one ormore of the tasks may be obtained. For example, the minimum set of UIelements required to complete the highest ranked task is obtained, theminimum set of UI elements required to complete second highest rankedtask is obtained, etc. The minimum set of UI elements for a task mayhave been previously determined using the process depicted in FIG. 2.

In Step 420, a custom UI for the highest ranked task is generated. Thecustom UI includes the minimum set of UI elements required to completethe task, but excludes the other UI elements of the application. Theminimum set of UI elements may be positioned in the custom UI based onthe ordering of the UI elements in the minimum set (discussed above).This custom UI may be transmitted to the user device.

In Step 425, a custom UI for the second highest ranked tasked isgenerated. The custom UI includes the minimum set of UI elementsrequired to complete the second highest ranked task, but excludes theother UI elements of the application. An image of the custom UI may begenerated. The image may have a low resolution in order to reduce thespace needed to store the image and/or reduce the network bandwidthneeded to transmit the image. However, the image may have a sufficientlyhigh resolution in order for the viewer (e.g., user) to be able toidentify the UI elements present in the custom UI. The image of thecustom UI may be transmitted to the user device for display to a user.

Step 425 may be repeated for the task with the third highest rank, thetask with the fourth highest rank, etc. up to some cutoff.

In one or more embodiments, while the custom UI associated with thehighest ranked task is displayed to the user on the user device, one ormore options may also be displayed to the user on the user device. Eachoption may correspond to a task other than the task with the highestrank. For example, the first option may correspond to the task with thesecond highest rank. The second option may correspond to the task withthe third highest rank. If the user selects the second option, the imageof the custom UI for the task with the third highest rank may bedisplayed. The image may be displayed with (e.g., on top of) orseparately from the custom UI associated with the highest rank task.

Those skilled in the art, having the benefit of this detaileddescription, will appreciate that the highest ranked task might not bethe task the user is attempting to complete. In other words, theprediction model's prediction was incorrect. By presenting theseoptions, the user can effectively preview other custom UIs and selectthe correct custom UI (i.e., the custom UI corresponding to the taskthat the user is actually trying to complete). The custom UIcorresponding to the correct task, as selected by the user, can then berequested (e.g. from the UI generator), generated, and/or displayed tothe user in order to complete the task. The selection by the user mayalso be used as feedback to update the prediction model.

Those skilled in the art, having the benefits of this detaileddescription, will also appreciate that by initially transmitting theimages instead of the actual custom UIs, less network bandwidth isrequired (as opposed to transmitting the custom UIs for all the tasks).

In one or more embodiments, a prompt may be displayed to the user on theuser device requesting confirmation that the task with the highest rankis indeed the task the user is attempting to complete. The prompt mightbe displayed to all users. Alternatively, the prompt might only bedisplayed to expert users. The prompt may be optional. The response tothe prompt, positive or negative, may be used as feedback to improveprediction model.

FIG. 5 shows an example of a custom UI for completing a task along withan image of another custom UI for completing a different task inaccordance with one or more embodiments. In FIG. 5, it is assumed thattask A had the highest rank. In other words, it was predicted that theuser operating the user device is attempting to complete task A. Asshown in FIG. 5, the custom UI for task A includes the minimum set of UIelements 510 required to complete task A. The custom UI for task A mayhave been generated by the UI generator 106 and transmitted to the userdevice (discussed above in reference to FIG. 4).

FIG. 5 also shows two options 520. The options 520 are displayed on topof the custom UI for task A. One option asks if the user is trying tocomplete task Q. Task Q might have had the second highest rank. Theother option asks if the user is trying to complete task Z. Task Z mighthave the third highest rank. By selecting the first option, the image ofthe custom UI for task Q 525 is displayed. This image 525 may have beengenerated by the UI generator 106 and transmitted to the user device(discussed above in reference to FIG. 4).

In the event the user is not actually trying to complete task A (i.e.,the prediction model was incorrect), the user is able to preview othercustom UIs (e.g., for task Q and task Z) and select the custom UI forthe task the user is actually trying to complete. The user maymanipulate the UI elements on the custom UI in order to complete thetask the user is actually trying to complete.

Various embodiments can be implemented, for example, using one or morecomputer systems, such as computer system 600 shown in FIG. 6. Computersystem 600 can be used, for example, to implement the processes shown inFIG. 2 and FIG. 4. For example, computer system 600 can be configured todetermine a minimum set of UI elements required to complete a task.Computer system 600 can also be configured to generate a custom UI forcompleting a predicted task. Computer system 600 can be any computercapable of performing the functions described herein.

Computer system 600 can be any well-known computer capable of performingthe functions described herein.

Computer system 600 includes one or more processors (also called centralprocessing units, or CPUs), such as a processor 604. Processor 604 isconnected to a communication infrastructure or bus 606.

One or more processors 604 may each be a graphics processing unit (GPU).In an embodiment, a GPU is a processor that is a specialized electroniccircuit designed to process mathematically intensive applications. TheGPU may have a parallel structure that is efficient for parallelprocessing of large blocks of data, such as mathematically intensivedata common to computer graphics applications, images, videos, etc. Oneor more GPUs may be specialized for executing the machine learning tasks(e.g., clustering, prediction, etc.) discussed above.

Computer system 600 also includes user input/output device(s) 603, suchas monitors, keyboards, pointing devices, etc., that communicate withcommunication infrastructure 606 through user input/output interface(s)602.

Computer system 600 also includes a main or primary memory 608, such asrandom access memory (RAM). Main memory 608 may include one or morelevels of cache. Main memory 608 has stored therein control logic (i.e.,computer software) and/or data.

Computer system 600 may also include one or more secondary storagedevices or memory 610. Secondary memory 610 may include, for example, ahard disk drive 612 and/or a removable storage device or drive 614.Removable storage drive 614 may be a floppy disk drive, a magnetic tapedrive, a compact disk drive, an optical storage device, tape backupdevice, and/or any other storage device/drive.

Removable storage drive 614 may interact with a removable storage unit618. Removable storage unit 618 includes a computer usable or readablestorage device having stored thereon computer software (control logic)and/or data. Removable storage unit 618 may be a floppy disk, magnetictape, compact disk, DVD, optical storage disk, and/any other computerdata storage device. Removable storage drive 614 reads from and/orwrites to removable storage unit 618 in a well-known manner.

According to an exemplary embodiment, secondary memory 610 may includeother means, instrumentalities or other approaches for allowing computerprograms and/or other instructions and/or data to be accessed bycomputer system 600. Such means, instrumentalities or other approachesmay include, for example, a removable storage unit 622 and an interface620. Examples of the removable storage unit 622 and the interface 620may include a program cartridge and cartridge interface (such as thatfound in video game devices), a removable memory chip (such as an EPROMor PROM) and associated socket, a memory stick and USB port, a memorycard and associated memory card slot, and/or any other removable storageunit and associated interface.

Computer system 600 may further include a communication or networkinterface 624. Communication interface 624 enables computer system 600to communicate and interact with any combination of remote devices,remote networks, remote entities, etc. (individually and collectivelyreferenced by reference number 628). For example, communicationinterface 624 may allow computer system 600 to communicate with remotedevices 628 over communications path 626, which may be wired and/orwireless, and which may include any combination of LANs, WANs, theInternet, etc. Control logic and/or data may be transmitted to and fromcomputer system 600 via communication path 626.

In an embodiment, a tangible, non-transitory apparatus or article ofmanufacture comprising a tangible, non-transitory computer useable orreadable medium having control logic (software) stored thereon is alsoreferred to herein as a computer program product or program storagedevice. This includes, but is not limited to, computer system 600, mainmemory 608, secondary memory 610, and removable storage units 618 and622, as well as tangible articles of manufacture embodying anycombination of the foregoing. Such control logic, when executed by oneor more data processing devices (such as computer system 600), causessuch data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, it will be apparentto persons skilled in the relevant art(s) how to make and useembodiments of this disclosure using data processing devices, computersystems and/or computer architectures other than that shown in FIG. 6.In particular, embodiments can operate with software, hardware, and/oroperating system implementations other than those described herein.

It is to be appreciated that the Detailed Description section, and notany other section, is intended to be used to interpret the claims. Othersections can set forth one or more but not all exemplary embodiments ascontemplated by the inventor(s), and thus, are not intended to limitthis disclosure or the appended claims in any way.

While this disclosure describes exemplary embodiments for exemplaryfields and applications, it should be understood that the disclosure isnot limited thereto. Other embodiments and modifications thereto arepossible, and are within the scope and spirit of this disclosure. Forexample, and without limiting the generality of this paragraph,embodiments are not limited to the software, hardware, firmware, and/orentities illustrated in the figures and/or described herein. Further,embodiments (whether or not explicitly described herein) havesignificant utility to fields and applications beyond the examplesdescribed herein.

Embodiments have been described herein with the aid of functionalbuilding blocks illustrating the implementation of specified functionsand relationships thereof. The boundaries of these functional buildingblocks have been arbitrarily defined herein for the convenience of thedescription. Alternate boundaries can be defined as long as thespecified functions and relationships (or equivalents thereof) areappropriately performed. Also, alternative embodiments can performfunctional blocks, steps, operations, methods, etc. using orderingsdifferent than those described herein.

References herein to “one embodiment,” “an embodiment,” “an exampleembodiment,” or similar phrases, indicate that the embodiment describedcan include a particular feature, structure, or characteristic, butevery embodiment can not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it would be within the knowledge of persons skilled in therelevant art(s) to incorporate such feature, structure, orcharacteristic into other embodiments whether or not explicitlymentioned or described herein. Additionally, some embodiments can bedescribed using the expression “coupled” and “connected” along withtheir derivatives. These terms are not necessarily intended as synonymsfor each other. For example, some embodiments can be described using theterms “connected” and/or “coupled” to indicate that two or more elementsare in direct physical or electrical contact with each other. The term“coupled,” however, can also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other.

The breadth and scope of this disclosure should not be limited by any ofthe above-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

What is claimed is:
 1. A method, comprising: transmitting, by one ormore computing devices, a plurality of user interface (UI) elementsassociated with an application to a first computing device operated by afirst user and a second computing device operated by a second user;obtaining, by the one or more computing devices, a plurality of datastructures comprising: a first data structure identifying a first subsetof the plurality of UI elements manipulated by the first user while thefirst user was completing a first task in the application using thefirst computing device; and a second data structure identifying a secondsubset of the plurality of UI elements manipulated by the second userwhile the second user was completing a second task in the applicationusing the second computing device; grouping, using a machine learningmodel executing on the one or more computing devices, the plurality ofdata structures into a plurality of clusters comprising: a first clustercorresponding to the first task and comprising the first data structure;and determining, using the one or more computing devices, a minimum setof UI elements required for the first task based on the first cluster.2. The method of claim 1, further comprising: determining, using one ormore computing devices, a minimum set of UI elements required for thesecond task based on a second cluster comprising the second datastructure.
 3. The method of claim 2, wherein: the first cluster furthercomprises a first subset of the plurality of data structures, the secondcluster further comprises a second subset of the plurality of datastructures, determining the minimum set of UI elements for the firsttask comprises executing a union operation on the first subset of theplurality of data structures, and determining the minimum set of UIelement for the second task comprises executing a union operation on thesecond subset of the plurality of data structures.
 4. The method ofclaim 3, further comprising: calculating a plurality of distancesbetween the plurality of clusters.
 5. The method of claim 4, furthercomprising: obtaining a first plurality of timestamps associated withthe first subset of the plurality of data structures; determining afirst age statistic for the first subset of the plurality of datastructures based on the first plurality of timestamps; obtaining asecond plurality of timestamps associated with the second subset of theplurality of data structures; and determining a second age statistic forthe second subset of the plurality of data structures based on thesecond plurality of timestamps.
 6. The method of claim 1, furthercomprising: generating a custom UI for the first task comprising theminimum set of UI elements for the first task but excluding other UIelements in the plurality of UI elements.
 7. The method of claim 1,wherein the plurality of UI elements comprise at least one selected froma group consisting of radio buttons, a textbox, and a drop-down list,and wherein the first task updates a field of a record.
 8. A system,comprising: one or more computer processors; and a memory storinginstructions that when executed by the one or more computer processorscause the one or more computer processors to: transmit a plurality ofuser interface (UI) elements associated with an application to a firstcomputing device operated by a first user and a second computing deviceoperated by a second user; obtain a plurality of data structurescomprising: a first data structure identifying a first subset of theplurality of UI elements manipulated by the first user while the firstuser was completing a first task in the application using the firstcomputing device; and a second data structure identifying a secondsubset of the plurality of UI elements manipulated by the second userwhile the second user was completing a second task in the applicationusing the second computing device; group, using a machine learningmodel, the plurality of data structures into a plurality of clusterscomprising: a first cluster corresponding to the first task andcomprising the first data structure; and determine a minimum set of UIelements required for the first task based on the first cluster.
 9. Thesystem of claim 8, wherein the instructions further cause the one ormore computer processors to: determine a minimum set of UI elementsrequired for the second task based on a second cluster comprising thesecond data structure.
 10. The system of claim 9, wherein: the firstcluster further comprises a first subset of the plurality of datastructures, the second cluster further comprises a second subset of theplurality of data structures, determining the minimum set of UI elementsfor the first task comprises executing a union operation on the firstsubset of the plurality of data structures, and determining the minimumset of UI element for the second task comprises executing a unionoperation on the second subset of the plurality of data structures. 11.The system of claim 10, wherein the instructions further cause the oneor more computer processors to: calculate a plurality of distancesbetween the plurality of clusters.
 12. The system of claim 11, whereinthe instructions further cause the one or more computer processors to:obtain a first plurality of timestamps associated with the first subsetof the plurality of data structures; determine a first age statistic forthe first subset of the plurality of data structures based on the firstplurality of timestamps; obtain a second plurality of timestampsassociated with the second subset of the plurality of data structures;and determine a second age statistic for the second subset of theplurality of data structures based on the second plurality oftimestamps.
 13. The system of claim 8, wherein the instructions furthercause the one or more computer processors to: generate a custom UI forthe first task comprising the minimum set of UI elements for the firsttask but excluding other UI elements in the plurality of UI elements.14. The system of claim 8, wherein the plurality of UI elements compriseat least one selected from a group consisting of radio buttons, atextbox, and a drop-down list.
 15. A non-transitory computer readablemedium (CRM) storing instructions, the instructions, when executed byone or more computer processors, cause the one or more computerprocessors to: transmit a plurality of user interface (UI) elementsassociated with an application to a first computing device operated by afirst user and a second computing device operated by a second user;obtain a plurality of data structures comprising: a first data structureidentifying a first subset of the plurality of UI elements manipulatedby the first user while the first user was completing a first task inthe application using the first computing device; and a second datastructure identifying a second subset of the plurality of UI elementsmanipulated by the second user while the second user was completing asecond task in the application using the second computing device; group,using a machine learning model, the plurality of data structures into aplurality of clusters comprising: a first cluster corresponding to thefirst task and comprising the first data structure; and determine aminimum set of UI elements required for the first task based on thefirst cluster.
 16. The non-transitory CRM of claim 15, wherein theinstructions further cause the one or more computer processors to:determine a minimum set of UI elements required for the second taskbased on a second cluster comprising the second data structure.
 17. Thenon-transitory CRM of claim 16, wherein: the first cluster furthercomprises a first subset of the plurality of data structures, the secondcluster further comprises a second subset of the plurality of datastructures, determining the minimum set of UI elements for the firsttask comprises executing a union operation on the first subset of theplurality of data structures, and determining the minimum set of UIelement for the second task comprises executing a union operation on thesecond subset of the plurality of data structures.
 18. Thenon-transitory CRM of claim 17, wherein the instructions further causethe one or more computer processors to: obtain a first plurality oftimestamps associated with the first subset of the plurality of datastructures; determine a first age statistic for the first subset of theplurality of data structures based on the first plurality of timestamps;obtain a second plurality of timestamps associated with the secondsubset of the plurality of data structures; and determine a second agestatistic for the second subset of the plurality of data structuresbased on the second plurality of timestamps.
 19. The non-transitory CRMof claim 15, wherein the instructions further cause the one or morecomputer processors to: generate a custom UI for the first taskcomprising the minimum set of UI elements for the first task butexcluding other UI elements in the plurality of UI elements.
 20. Thenon-transitory CRM of claim 15, wherein the plurality of UI elementscomprise at least one selected from a group consisting of radio buttons,a textbox, and a drop-down list.